Imaging device that generates multiple-exposure image data

ABSTRACT

An imaging device including a pixel, and a controller that sets a sensitivity of the pixel according to an external signal, where the external signal includes sound, vibration or inclination.

CROSS-REFERENCE OF RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.16/782,411, which is a Continuation of U.S. patent application Ser. No.15/497,193, filed on Apr. 25, 2017, now U.S. Pat. No. 10,594,948, whichis a Continuation of International Application No. PCT/JP2016/002945,filed Jun. 20, 2016, which in turn claims the benefit of JapaneseApplication No. 2015-138393, filed on Jul. 10, 2015, the entiredisclosures of which Applications are incorporated by reference herein.

BACKGROUND 1. Technical Field

The present disclosure relates to an imaging device, and particularly toan imaging device capable of performing a multiple exposure.

2. Description of the Related Art

Image capturing systems are widely used in the field of consumer goods,industrial goods, and the like. Such image capturing systems typicallyinclude imaging devices that perform a single exposure (may be referredto a “typical exposure”) and a single reading during a single frameperiod. Imaging devices that perform a plurality of exposures and asingle reading during a single frame period are also known. Theplurality of exposures are typically referred to as a “multipleexposure”. In a multiple exposure, image data that are obtained bysequentially capturing images are superimposed, and the superimposedimage data are read as one image data.

Image data obtained with a multiple exposure include information on thechange of a subject, such as motion information. In a multiple exposure,an image of a stationary subject, which does not move, such as thebackground, is captured in the same manner as in a typical exposure. Onthe other hand, in the case where an image of a moving subject iscaptured, the change of the moving subject can be reflected on the imagedata obtained with a multiple exposure. With a multiple exposure, thelocus of the motion of the subject can be observed in a single image. Inthis manner, a multiple exposure is useful for analyzing moving subjectsand for analyzing high-speed phenomena.

Japanese Unexamined Patent Application Publication No. 2001-197373discloses an imaging device capable of capturing images with a multipleexposure (hereinafter referred to as “multiple-exposure imaging”) whilechanging the exposure period. With the imaging device, a more naturalseries of images can be obtained at a high dynamic resolution.

SUMMARY

In the above-described imaging device of the related art, a furtherimprovement has been required for imaging a moving subject.

In one general aspect, the techniques disclosed here feature an imagingdevice including a pixel, and a controller that sets a sensitivity ofthe pixel according to an external signal, where the external signalincludes sound, vibration or inclination.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a storage medium, or any selective combination thereof.

According to an aspect of the present disclosure, it is possible toprovide an imaging device that can further improve an image of a movingsubject.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram schematically illustrating the configurationof an imaging device according to a first exemplary embodiment;

FIG. 2A is a block diagram schematically illustrating a typicalconfiguration of an image obtaining unit;

FIG. 2B schematically illustrates a cross-sectional view of a unit pixelcell;

FIG. 3 illustrates a typical operation timing of a multiple exposureduring a single frame period;

FIG. 4 schematically illustrates image data obtained with amultiple-exposure imaging;

FIG. 5 is a timing chart illustrating how a potential difference betweena first electrode and a second electrode in a multiple exposure during asingle frame period changes (how the voltage applied to the firstelectrode changes);

FIG. 6 schematically illustrates an image of multiple-exposure imagedata and images of individually separated image data;

FIG. 7 schematically illustrates images of individual image data inwhich the levels of a common display attribute are all corrected to bethe maximum value;

FIG. 8A illustrates an image of multiple-exposure image data with asuperimposed indicator;

FIG. 8B illustrates an image of multiple-exposure image data with asuperimposed indicator;

FIG. 9A schematically illustrates an image of image data that areobtained by processing the image data in FIG. 8A to have the same levelof a common display attribute;

FIG. 9B schematically illustrates an image of image data that areobtained by processing the image data in FIG. 8B to have the same levelof a common display attribute;

FIG. 10 is a block diagram schematically illustrating the configurationof an imaging device according to a second exemplary embodiment;

FIG. 11 is a timing chart illustrating a typical operation timing of amultiple exposure during a single frame period;

FIG. 12A schematically illustrates an exemplary image obtained with amultiple exposure;

FIG. 12B schematically illustrates an exemplary image obtained with amultiple exposure;

FIG. 13A is a timing chart illustrating a typical operation timing of amultiple exposure during a single frame period;

FIG. 13B schematically illustrates an exemplary image obtained with amultiple exposure; and

FIG. 14 is a block diagram schematically illustrating the configurationof an imaging device according to a fourth exemplary embodiment.

DETAILED DESCRIPTION

Prior to description of embodiments of the present disclosure, issues ofthe related art that the present inventor has researched will bedescribed.

For example, Japanese Unexamined Patent Application Publication No.2001-197373 proposes a method for changing brightness of subject imagesin a time-series manner by changing an exposure period for each imagecapture of a multiple exposure. However, the exposure period is closelyrelated to the speed of a subject. Accordingly, the usable exposureperiod is restricted in order to obtain a blurless image. In addition,since the exposure interval is not constant, it is not possible tocapture images that are perfectly corresponding to the time base.

In view of such issues, the present inventor has arrived at an imagingdevice having a novel configuration.

The present disclosure encompasses imaging devices described in thefollowing items.

Item 1

An imaging device including an image obtaining unit that obtains aplurality of image data at a plurality of timings during a single frameperiod and that multiplexes the plurality of image data, the pluralityof image data including at least two image data having different levelsof a common display attribute, and an image output unit including anoutput interface that outputs the image data multiplexed by the imageobtaining unit.

With the imaging device according to Item 1, it is possible to determineindividual image data in a time-series manner in the multiple-exposureimage data.

Item 2

An imaging device including an image obtaining unit that obtains aplurality of image data at a plurality of timings during a single frameperiod and that multiplexes the plurality of image data, the pluralityof image data including at least two image data having different levelsof a common display attribute, and

-   an image outputting unit including an output interface that    demultiplexes, on the basis of subject information, the image data,    which have been multiplexed by the image obtaining unit, into the    image data related to each of the plurality of image data and that    individually outputs at least one of the demultiplexed image data.

With the imaging device according to Item 2, it is possible to determineindividual image data in a time-series manner in the multiple-exposureimage data. For example, from the multiplexed image data obtained duringa single frame period, moving-image data such as a frame-by-frameplayback moving image can be obtained.

Item 3

The imaging device according to Item 1 or 2, wherein the levels of thecommon display attribute differ between each data of the plurality ofimage data.

Item 4

The imaging device according to any one of Items 1 to 3, wherein thecommon display attribute is at least one of brightness and color.

With the imaging device according to Item 4, the common displayattribute can have variations.

Item 5

The imaging device according to any one of Items 1 to 4, wherein thetime-series change in the level of the common display attribute is amonotonic increase or a monotonic decrease.

Item 6

The imaging device according to any one of Items 1 to 4, wherein thetime-series change in the level of the common display attribute israndom.

Item 7

The imaging device according to any one of Items 1 to 6, wherein theimage output unit superimposes an indicator on the multiplexed imagedata, the indicator indicating a temporal transition during which theplurality of image data are obtained.

With the imaging device according to Item 7, since the indicator isadded in addition to the difference in the level of the displayattribute, it is easy to determine individual image data in a timeseries manner in the multiple-exposure image data.

Item 8

An imaging device including an image obtaining unit that obtains aplurality of image data at a plurality of timings during a single frameperiod and that multiplexes the plurality of image data, and

-   an image outputting unit that superimposes an indicator on the image    data multiplexed by the image obtaining unit, the indicator    indicating a temporal transition during which the plurality of image    data are obtained and that includes an output interface that outputs    the superimposed image data.

With the imaging device according to Item 8, it is possible to determineindividual image data in a time-series manner in the multiple-exposureimage data.

Item 9

The imaging device according to any one of Items 1 to 8, wherein theimage obtaining unit obtains, with a multiple exposure, the plurality ofimage data at a plurality of timings during a single frame period, andwherein all exposure periods in the multiple exposure are equal.

Item 10

The imaging device according to any one of Items 1 to 9, furtherincluding a display that displays an image on the basis of the imagedata output from the image output unit.

With the imaging device according to Item 10, it is possible toimmediately observe the captured image by the display displaying thecaptured image.

Item 11

The imaging device according to any one of Items 1 to 10, wherein theimage obtaining unit includes a pixel array in which unit pixel cellsare two-dimensionally arrayed,

-   in which each of the unit pixel cells-   includes    -   a photoelectric converter that includes a first electrode, a        second electrode, and a photoelectric conversion film between        the first electrode and the second electrode and that performs        photoelectric conversion on incident light, and    -   a charge detection circuit that is connected to the second        electrode and that detects a signal charge generated by the        photoelectric converter, and-   in which the image obtaining unit obtains, with a global shutter and    a multiple exposure, the plurality of image data at the plurality of    timings during a single frame period.

With the imaging device according to Item 11, it is possible to providethe imaging device including an image sensor that includes thephotoelectric converter.

Item 12

The imaging device according to Item 11, wherein the image obtainingunit changes a potential difference between the first electrode and thesecond electrode for each image capture with a multiple exposure.

With the imaging device according to Item 12, since the potentialdifference between the first electrode and the second electrode ischanged, the sensitivity of each of the unit pixel cells per unit timecan be changed.

Item 13

The imaging device according to Item 11, in which the sensitivity ofeach of the unit pixel cells per unit time is changed for each imagecapture with a multiple exposure.

Item 14

An imaging device comprising a unit pixel cell, wherein

-   the unit pixel cell captures first data in a first exposure period    and captures second data in a second exposure period different from    the first exposure period, the first exposure period and the second    exposure period being included in a frame period,-   a sensitivity per unit time of the unit pixel cell in the first    exposure period is different from a sensitivity per unit time of the    unit pixel cell in the second exposure period, and-   the imaging device outputs multiple-exposure image data including at    least the first data and the second data.

Item 15

The imaging device according to Item 14, wherein

-   the unit pixel cell includes    -   a photoelectric converter including a first electrode, a second        electrode facing the first electrode, and a photoelectric        conversion film between the first electrode and the second        electrode, the photoelectric converter generating signal charge        through photoelectric conversion, and    -   a signal detection circuit electrically connected to the second        electrode, the signal detection circuit detecting the signal        charge, and-   a potential difference between the first electrode and the second    electrode in the first exposure period is different from a potential    difference between the first electrode and the second electrode in    the second exposure period.

Item 16

The imaging device according to Item 15, further comprising a voltagecontrol circuit that supplies a first voltage to the first electrode inthe first exposure period and supplies a second voltage to the firstelectrode in the second exposure period, the first voltage beingdifferent from the second voltage.

Item 17

The imaging device according to Item 16, wherein the voltage controlcircuit supplies a third voltage to the first electrode in anon-exposure period provided between the first exposure period and thesecond exposure period such that the signal detection circuit does notdetect the signal charge.

Item 18

The imaging device according to one of Item 14 to 17, wherein a lengthof the first exposure period is the same as a length of the secondexposure period.

Item 19

The imaging device according to one of Item 14 to 18, wherein

-   the unit pixel cell captures, in the frame period, a plurality of    data in a plurality of exposure periods including the first and    second exposure periods, the plurality of exposure periods being    different from each other, and-   a sensitivity of the unit pixel cell per unit time in an earlier    exposure period among the plurality of exposure periods is higher    than a sensitivity of the unit pixel cell per unit time in a later    exposure period among the plurality of exposure periods.

Item 20

The imaging device according to one of Item 14 to 18, wherein

-   the unit pixel cell captures, in the frame period, a plurality of    data in a plurality of exposure periods including the first and    second exposure periods, the plurality of exposure periods being    different from each other, and-   a sensitivity of the unit pixel cell per unit time in an earlier    exposure period among the plurality of exposure periods is lower    than a sensitivity of the unit pixel cell per unit time in a later    exposure period among the plurality of exposure periods.

Item 21

The imaging device according to one of Item 14 to 18, wherein

-   the unit pixel cell captures, in the frame period, a plurality of    data in a plurality of exposure periods including the first and    second exposure periods, the plurality of exposure periods being    different from each other, and-   a sensitivity of the unit pixel cell per unit time in each of the    plurality of exposure periods is set at random.

Item 22

The imaging device according to one of Item 14 to 18, further comprisinga signal processing circuit that extracts, from the multiple-exposureimage data, a first subject image based on the first data and a secondsubject image based on the second data, the first subject image and thesecond subject image corresponding to a moving subject.

Item 23

The imaging device according to Item 22, wherein brightness or colors ofthe first subject image is different from brightness or colors of thesecond subject image.

Item 24

The imaging device according to Item 23, wherein the signal processingcircuit performs a process for correcting, on the basis of the firstsubject image and the second subject image, brightness or colors of thefirst subject image and the second subject image in themultiple-exposure image data to be same.

Item 25

The imaging device according to Item 22, wherein the signal processingcircuit performs a process for adding, on the basis of the first subjectimage and the second subject image, an indicator indicating a temporaltransition to each of the first subject image and the second subjectimage in the multiple-exposure image data.

According to an aspect of the present disclosure, it is possible todetermine the time series of individual image data in the multiplexedimage data.

Now, embodiments of the present disclosure will be described below withreference to the drawings. It should be noted that the presentdisclosure is not limited to the following embodiments. In addition,modification may be made as appropriate without departing from the scopeof the effects of the present disclosure. Furthermore, an embodiment maybe combined with another embodiment. In the following description, thesame or similar components are denoted by the same reference numerals,and a repeated description may be omitted. It should also be noted thatan exemplary pulse wave may have an arbitrarily set pulse interval,pulse width, pulse amplitude, and phase relationship with another signaland does not have to have the illustrated ones.

First Embodiment 1.1. Configuration of Imaging Device 100

FIG. 1 is a block diagram schematically illustrating the configurationof an imaging device 100 according to a first embodiment. FIG. 2A is ablock diagram schematically illustrating a typical configuration of animage obtaining unit 120. FIG. 2B schematically illustrates across-sectional view of each unit pixel cell 121.

The imaging device 100 includes an optical system 110, the imageobtaining unit 120, an image output unit 130, and a system controller140.

The optical system 110 includes a diaphragm, an image-stabilizer lens, azoom lens, a focal lens, and the like. A subject image can be enlargedand reduced by moving the zoom lens along the optical axis. In addition,the focal position of the subject image can be adjusted by moving thefocal lens along the optical axis. Note that the appropriate number oflenses included in the optical system 110 is determined in accordancewith required functions.

The image obtaining unit 120 is a so-called image sensor. As illustratedin FIG. 2A, for example, the image obtaining unit 120 includes a pixelarray in which a plurality of unit pixel cells 121 are two-dimensionallyarrayed and peripheral driving circuits. The peripheral driving circuitsinclude a row scanning circuit 122 and a column scanning circuit 123.The plurality of unit pixel cells 121 are electrically connected to therow scanning circuit 122 and the column scanning circuit 123 viarespective signal lines. The peripheral driving circuits further includea voltage control circuit 124. The voltage control circuit 124 will bedescribed later.

As illustrated in FIG. 2B, each of the unit pixel cells 121 includes aphotoelectric converter 121D for photoelectric conversion of incidentlight and a charge detecting circuit 121F. The photoelectric converter121D includes a first electrode 121A, a second electrode 121B, and aphotoelectric conversion film 121C between the first electrode 121A andthe second electrode 121B. The charge detecting circuit 121F is providedon a semiconductor substrate 121H and is electrically connected to thesecond electrode 121B via a contact plug 121E in an interlayerinsulating layer 121G. The charge detecting circuit 121F can detect asignal charge generated by the photoelectric converter 121D. Note thatthe unit pixel cell 121 is typically a stack cell including aphotoelectric conversion film but may be a cell including a siliconphotodiode.

The image obtaining unit 120 includes a control circuit that changes thelevel of a common display attribute on the basis of a control signalindicating the level of the common display attribute, that is, a controlsignal corresponding to the level of the common display attribute inimage data to be obtained. The image obtaining unit 120 obtains, at aplurality of timings during a single frame period, a plurality of imagedata having different levels of the common display attribute andmultiplexes the plurality of image data. That is, during a single frameperiod, the photoelectric converter 121D obtains, at a plurality oftimings, a plurality of image data having different levels of the commondisplay attribute. The charge detecting circuit 121F readscaptured-image data including the above plurality of image data. Thecharge detecting circuit 121F may individually read the plurality ofimage data. In this case, the plurality of image data are read by thecharge detecting circuit 121F and are then superimposed. The commondisplay attribute will be described later.

The image output unit 130 includes an output buffer 131. The imageoutput unit 130 outputs the captured-image data through the outputbuffer 131 to the outside. The image output unit 130 can output, to theoutside, the image data multiplexed by the image obtaining unit 120.

The data output from the image output unit 130 is typically raw data andis, for example, a 12-bit signal. However, the present disclosure is notlimited to this example, and the output data may be, for example,compressed data that is compressed in a manner complying with the H.264standard, for example. The compressed data is, for example, an 8-bitsignal. In this case, for example, the image output unit 130 may includea video codec that generates the compressed data in a manner complyingwith the H.264 standard.

The system controller 140 controls the overall imaging device 100.Specifically, the system controller 140 controls the optical system 110,the image obtaining unit 120, and the image output unit 130. The systemcontroller 140 is typically a semiconductor integrated circuit, and is,for example, a central processing unit (CPU).

For example, the imaging device 100 may be connected to an externaldevice 200 including a signal processing circuit 210, a display 220, andthe like. Examples of the external device 200 include a personalcomputer, a smartphone, and the like. Specifically, the imaging device100 is electrically connected to the signal processing circuit 210, suchas a digital signal processor (DSP). The signal processing circuit 210receives signals output from the imaging device 100 and performs aprocess such as gamma correction, color interpolation, spaceinterpolation, or automatic white balance. The display 220 is, forexample, a liquid crystal display or an organic electroluminescent (EL)display and displays an image on the basis of signals output from thesignal processing circuit 210. Note that data output from the imagingdevice 100 may temporarily be stored in a recording medium such as aflash memory.

1.2. Operation of Imaging Device 100

FIG. 3 illustrates a typical operation timing of a multiple exposureduring a single frame period. VD in FIG. 3 denotes a start pulse of theframe period. An exposure signal is a signal indicating whether anexposure is valid or invalid. High Period is a period during which theexposure is valid (exposure state), and Low Period is a period duringwhich the exposure is invalid (non-exposure state). A display-attributesetting signal is a signal for setting the level of a common displayattribute. The higher the set signal pulse is, the higher the level ofthe common display attribute is. Note that a single signal may serve asboth the exposure signal and the display-attribute setting signal.

The term “common display attribute” herein refers to at least one ofbrightness and color. The term “brightness” means the brightnessobtained on the basis of RGB signals in each pixel cell, and the term“color” means a hue or chroma obtained on the basis of RGB signals.

The pulse waveform of the exposure signal and that of thedisplay-attribute setting signal respectively indicate the exposureperiod and the level of the common display attribute. Each of the pulsewaveforms, that is, a control signal, is supplied to the image obtainingunit 120. The control signal may be a control signal generated by thesystem controller 140 or may be a control signal generated inside theimage obtaining unit 120.

FIG. 3 illustrates an example in which five exposures are performedduring a single frame period. The image obtaining unit 120 multiplexes aplurality of image data obtained through each image capture. In otherwords, one image data is obtained by superimposing the plurality ofimage data. An image of a subject without motion, such as thebackground, is captured in each exposure period by unit pixel cells 121located at the same positions in the pixel array. Then, the image dataare accumulated by the same unit pixel cells 121. On the other hand, animage of a subject with motion between respective image captures iscaptured by unit pixel cells 121 in the pixel array where the subjectimages are focused at the respective exposure timings. As a result, ifdifferent unit pixel cells 121 have been involved in image capture inall of the five exposures, the respective image data are superimposed asfive independent images.

By performing each image capture by changing the level of a commondisplay attribute in accordance with the display-attribute settingsignal, it is possible to change the captured-image data obtained ineach exposure period. For example, through image captures by changingthe brightness of captured-image data for each image capture, it ispossible to obtain a plurality of image data having differentbrightness. Accordingly, in the superimposed image data, the levels ofthe common display attribute are different between five images of thesubject with motion.

FIG. 4 schematically illustrates the image data obtained with themultiple-exposure imaging illustrated in FIG. 3. The differences in thelevel of the common display attribute are individually expressed bybrightness. The higher the brightness is, the later the image data is inthe time series. In this manner, the time series of the obtained imagedata can be determined on the basis of brightness information.

The manner in which the level of the common display attribute changes isarbitrarily set (or the regularity of change is arbitrarily set). Theregularity of change is determined upon starting the system, forexample. Since the regularity of change is already known, it is possibleto determine the time series of the images of the same subject withmotion on the basis of the image data obtained with a multiple exposurein addition to the above-described example. The time-series change inthe level of the common display attribute is typically a monotonicincrease or a monotonic decrease. Note that the time-series change inthe level of the common display attribute may be random.

As illustrated in FIG. 3, in this embodiment, all of the exposureperiods for a multiple exposure are typically equal, and all of theexposure intervals are also equal during a single frame period. Thispoint is markedly different from the related art in which the exposureperiod is adjusted. However, as a matter of fact, each exposure periodmay be changed as in the related art.

In this embodiment, instead of changing the exposure period, thesensitivity per unit time of a unit pixel cell 121 is changed for eachimage capture with a multiple exposure. As a result, it is possible tochange the brightness and color information for each image capture.Specifically, by changing the potential difference between the firstelectrode 121A and the second electrode 121B of the photoelectricconverter 121D, it is possible to change the sensitivity per unit time(see FIG. 2B). Such change in the sensitivity is specifically describedin Japanese Unexamined Patent Application Publication No. 2007-104113,for example, the entire contents of which are herein incorporated byreference.

The manner in which the sensitivity in image captures is changed isarbitrarily set (or the regularity of change is arbitrarily set). Thetime-series change in the sensitivity is typically a monotonic increaseor a monotonic decrease. Note that the time-series change in thesensitivity may be random.

In addition, by decreasing the potential difference between the firstelectrode 121A and the second electrode 121B, it is possible to preventa charge obtained through photoelectric conversion from being detected,thereby making the sensitivity substantially zero. Accordingly, a globalshutter operation can be achieved.

As described above, the image obtaining unit 120 includes a controlcircuit that changes the level of the common display attribute on thebasis of a control signal indicating the level of the common displayattribute. For example, the control circuit may be the voltage controlcircuit 124 (see FIG. 2A) that controls the voltage to be applied to thefirst electrode 121A and the second electrode 121B. The voltage controlcircuit 124 may be provided on the semiconductor substrate 121H or on asubstrate other than the semiconductor substrate 121H. Each of theplurality of unit pixel cells 121 is electrically connected to thevoltage control circuit 124 via a correspondingphotoelectric-conversion-film control line. Specifically, the potentialof the second electrode 121B is equal to the potential of the chargedetecting circuit 121F, and the photoelectric-conversion-film controlline is electrically connected to the first electrode 121A. The voltagecontrol circuit 124 supplies a predetermined potential to the firstelectrode 121A on the basis of the control signal indicating the levelof the common display attribute.

FIG. 5 schematically illustrates how the potential difference betweenthe first electrode 121A and the second electrode 121B in a multipleexposure during a single frame period changes (how the voltage appliedto the first electrode 121A changes). In the example illustrated in FIG.5, the level of the common display attribute corresponds to thepotential difference between the first electrode 121A and the secondelectrode 121B. The higher a signal pulse for setting the potentialdifference is, the higher the potential difference is.

The Low level of the potential difference illustrated in FIG. 5corresponds to a level at which photoelectric conversion is notperformed. In other words, the Low level corresponds to a level at whicha global shutter operation is implementable. The levels other than theLow level correspond to levels that are sufficiently high to performphotoelectric conversion. The High level corresponds to a level at whichphotoelectric conversion is performed most efficiently. Accordingly, thehigher the potential difference is, the higher the sensitivity of theunit pixel cell 121 is. By repeating a cycle including the Low level andthe other potential difference levels a plurality of times, amultiple-exposure imaging is performed. By changing the potentialdifference for each image capture, the sensitivity in each exposureperiod is changed.

In this manner, by changing the potential difference between the firstelectrode 121A and the second electrode 121B for each image capture witha multiple exposure and by performing a global shutter operation, it ispossible to achieve both the multiple exposure and the change in thesensitivity for each image capture. As a result, for each image captureof a multiple exposure, it is possible to independently change the levelof the common display attribute (specifically the brightness). Sinceimages having different brightness are obtained through each imagecapture of a multiple exposure, it is possible to observe the timeseries of the images of the subject with motion in a single image dataobtained by multiplexing the image data.

As described above, it is also possible to set the color as the commondisplay attribute. In a stacked image sensor including a photoelectricconversion film, it is possible to change spectral characteristics bychanging the voltage to be applied to the photoelectric conversion film.For example, according to Japanese Unexamined Patent ApplicationPublication No. 2007-104113, by increasing and decreasing the voltage tobe applied to the photoelectric conversion film corresponding to anycolor of R, G, and B, it is possible to increase and decrease thesensitivity for each color. Such a feature is applicable to the presentdisclosure. For example, in the case where the subject is white, byperforming control in such a manner that the sensitivity for respectivecolors are reduced in the order of blue and green, it is possible tochange the color of the subject in such a manner that the colortemperature decreases in the order of white, yellow, and red.

For example, Japanese Unexamined Patent Application Publication No.2009-005061 discloses a technique to change spectral characteristics bychanging the bias voltage to the substrate in an image sensor includinga silicon photodiode to change a charge capturing region. With thistechnique, if the bias voltage increases, the sensitivity for light on along-wavelength side decreases accordingly. Such a feature is applicableto the present disclosure. For example, if the subject is white, byincreasing the bias voltage, it is possible to change the color of thesubject in such a manner that the color temperature increases in theorder of white, light blue, and blue.

In the above manner, by using various known techniques, the color can beset as the common display attribute. By dynamically changing spectralcharacteristics of the unit pixel cell 121 for each image capture, it ispossible to obtain a plurality of image data having different levels ofa common display attribute, that is, color, at a plurality of timingsduring a single frame period.

Next, another embodiment of the image output unit 130 will be described.The image output unit 130 may extract images of the subject with motionfrom the multiple-exposure image data obtained by the image obtainingunit 120 and may separate the images into individual image data.

FIG. 6 schematically illustrates the images of multiple-exposure imagedata and images of individually separated image data. The image dataillustrated in FIG. 6(a) correspond to the multiple-exposure image dataillustrated in FIG. 4.

The image obtaining unit 120 obtains a plurality of image data havingdifferent levels of the common display attribute at a plurality oftimings during a single frame period and multiplexes the plurality ofimage data. The image output unit 130 can detect patterns of images ofthe same subject on the basis of subject information from themultiple-exposure image data. Examples of the subject informationinclude patterns regarding the shape, brightness, and color of thesubject, and the like. The image output unit 130 can demultiplex themultiplexed image data into individual image data on the basis of thepattern-detection results. Then, the image output unit 130 can arrangethe separated individual image data in a time series manner on the basisof information on the change in the level of the common displayattribute. For example, the output buffer 131 can individually output atleast one of the separated image data in the order of the time series.

With such a configuration, it is possible to generate a plurality ofimage data each having a time order relationship on the basis of thesingle multiple-exposure image data. In the example illustrated in FIG.6, an image data including five images of a circular subject illustratedin FIG. 6(a) can be separated into five individual image dataillustrated in FIG. 6(b). By arranging the separated image data in atime series manner on the basis of the brightness information of each ofthe image data, a series of images such as a moving image can beobtained. In other words, from multiple-exposure image data obtainedduring a single frame period, data of a moving image such as aframe-by-frame playback moving image can be obtained.

By transmitting the moving image data to the external device 200, forexample, the external device 200 can play back the data of a movingimage, and the display 220 can display the frame-by-frame playbackmoving image. At the same time, by multiplexing the image data, it iseffective in compressing the data amount to be output from the imageobtaining unit 120. In addition, by performing a multiple-exposureimaging at a high speed, it is possible to obtain multiple-exposureimage data including a plurality of data obtained at extremely shorttime intervals. Accordingly, it is also possible to play back, as aslow-motion image, a phenomenon during a period in which an imagecapture is difficult.

In addition, the image output unit 130 may process the plurality ofimage data obtained in a time series manner so as to make the levels ofthe common display attribute equal in order to increase the visibility.If images of the same subject with motion in multiple-exposure imagedata are separated into a plurality of image data and are arranged in atime series manner, the time order relationship of the image dataobtained through each image capture becomes clear. For example, by usingknown information on the change in the level of the common displayattribute, the image output unit 130 may correct all of the levels ofthe common display attribute of individual image data to be the maximumvalue. Note that all of the levels of the common display attribute ofindividual image data may be corrected to be the average value, forexample, instead of the maximum value. Accordingly, the common displayattribute does not change frame by frame among individual subjectimages, thereby increasing the visibility.

FIG. 7 schematically illustrates images of individual image data inwhich the levels of the common display attribute are all corrected to bethe maximum value. Since the images of the circular subject amongindividual image data have the same brightness, if the image data areplayed back as a moving image, the impression of the moving image isimproved.

Still another embodiment of the image output unit 130 will be described.The image output unit 130 can superimpose, on the multiple-exposureimage data, an indicator indicating a temporal transition during whichthe image obtaining unit 120 obtained the plurality of image data.

FIGS. 8A and 8B illustrate how the indicator is superimposed on themultiplexed image data. As illustrated in FIG. 8A, the image output unit130 can determine the time series on the basis of information on thechange in the level of the common display attribute and can superimposean arrow indicating the temporal transition on the multiplexed imagedata. Alternatively, the image output unit 130 can add numeralsrepresenting the temporal transition near the subject images. Indicatorsare not limited to the examples illustrated in FIGS. 8A and 8B and maybe alphanumeric characters, signs such as arrows, or symbols such ascircles or triangles.

With such a configuration, since the indicator is added in addition tothe difference in the level of the display attribute, it is easy todetermine the time series of individual image data in themultiple-exposure image data.

The image output unit 130 may superimpose an indicator on themultiple-exposure image data, the indicator indicating the temporaltransition during which the image obtaining unit 120 obtained theplurality of image data and may further process the plurality of imagedata obtained in a time series manner to have the same level of thecommon display attribute. Accordingly, the visibility can be furtherincreased.

FIG. 9A schematically illustrates images of image data that are obtainedby processing the image data illustrated in FIG. 8A to have the samelevel of the common display attribute. FIG. 9B schematically illustratesimages of image data that are obtained by processing the image dataillustrated in FIG. 8B to have the same level of the common displayattribute.

It is sufficiently possible to determine the time series of the imagesillustrated in FIGS. 8A and 8B. Note that if the brightness differs foreach image capture with a multiple exposure, the visibility of a subjectimage with a low brightness inevitably decreases. Accordingly, the imageoutput unit 130 can correct all of the levels of the common displayattribute of individual image data to be the maximum value by usingknown information on the change in the level of the common displayattribute.

With such a configuration, the indicator makes it easier to determineindividual image data in a time-series manner in the multiple-exposureimage data, and further, the common display attribute does not changeamong individual subject images, thereby increasing the visibility.

According to this embodiment, it is possible to determine the time orderrelationship of images of a subject with motion or to determine asubject image that was obtained at a certain timing in multiple-exposureimage data obtained during a single frame period. For example, byconnecting the imaging device 100 to the external device 200, thedisplay 220 can display the subject images on the basis ofmultiple-exposure image data or each of the plurality of image data.

Second Embodiment

An imaging device 101 according to a second embodiment is different fromthe imaging device 100 according to the first embodiment in that theimaging device 101 includes an external-signal detecting unit 150. Thesame parts as those of the imaging device 100 will be omitted from thefollowing description, and most of the description refers to differentparts.

2.1. Imaging Device 101

FIG. 10 is a block diagram schematically illustrating the configurationof the imaging device 101 according to the second embodiment.

The imaging device 101 includes the optical system 110, the imageobtaining unit 120, the image output unit 130, the system controller140, and the external-signal detecting unit 150.

The external-signal detecting unit 150 detects an external signalregarding the external sound, light, vibration, inclination, and thelike. The external-signal detecting unit 150 transmits adisplay-attribute setting signal to the system controller 140 inaccordance with the external signal. The display-attribute settingsignal may be a binary signal of High and Low or may be an analog signalwhose value changes consecutively within a fixed range. Examples of theexternal-signal detecting unit 150 include a photodetector, amicrophone, and the like.

For example, in the case of capturing an image of a moving object thatemits a sound in accordance with the internal state thereof, amicrophone, which is a sound pressure detector, can be used as theexternal-signal detecting unit 150. As a typical mechanism, themicrophone converts the sound pressure into a voltage. The conversioncharacteristic is expressed in units of mV/Pa, for example, and theoutput voltage changes linearly with respect to the sound pressure.

For example, by referring to a lookup table in which the output voltageobtained from the microphone and a display-attribute setting signallevel are associated with each other, the external-signal detecting unit150 can generate a display-attribute setting signal. Alternatively, theexternal-signal detecting unit 150 may transmit an external signal tothe system controller 140, and the system controller 140 may refer tothe lookup table to determine the level of the display-attribute settingsignal in accordance with the external signal.

2.2. Operation of Imaging Device 101

FIG. 11 illustrates a typical operation timing of a multiple exposureduring a single frame period. The external signal indicates a signallevel detected by the external-signal detecting unit 150. In the case ofusing a microphone, the external signal is the above-described soundpressure. FIG. 11 illustrates discrete signal levels obtained bydetecting the degree of the sound emitted from the moving object. As inthe first embodiment, all of the exposure periods in the multipleexposure are equal, and all of the exposure intervals are also equal.

From the waveform of the external signal, it is found that the state ofthe moving object keeps changing. Time t1 to time t5 in FIG. 11 eachdenote a timing at which the level of the display-attribute settingsignal is determined at each image capture with a multiple exposure. Forexample, the system controller 140 performs sampling of the externalsignal level before each image capture of a multiple exposure anddetermines the setting level of the display attribute. Each imagecapture is performed in accordance with the determined setting level ofthe display attribute.

FIGS. 12A and 12B each illustrate exemplary images obtained with amultiple exposure. The differences in the level of a common displayattribute are individually expressed by brightness. That is, thedifferences in the degree of the sound are observed as the differencesin brightness by changing the sensitivity of each unit pixel cell 121.FIG. 12A illustrates exemplary images of image data obtained with amultiple-exposure imaging of the related art, and FIG. 12B illustratesexemplary images of image data obtained with a multiple-exposure imagingby using the imaging device 101 according to this embodiment. In therelated art, although the state of the moving object changed, there areno differences by which the change in the state can be observed in eachsubject image. In contrast, in the multiple-exposure imaging accordingto this embodiment, the brightness changes depending on the degree ofthe sound. Thus, the state change of the subject can be known by seeingthe differences in each subject image.

According to this embodiment, changes of light, sound, vibration or thelike, which are emitted from a subject, particularly from a movingobject, and affects the exterior, can be observed from the differencesin the level of the display attribute for each state.

Third Embodiment

An imaging device according to this embodiment is different from theimaging device 100 according to the first embodiment in that the imagingdevice changes the display-attribute setting signal at a specifictiming. The imaging device according to this embodiment has the sameconfiguration as the imaging device 100 according to the firstembodiment, the block diagram of which is illustrated in FIG. 1.

FIG. 13A illustrates a typical operation timing of a multiple exposureduring a single frame period. In the first embodiment, the level of thedisplay-attribute setting signal is changed for each image capture of amultiple exposure. In this embodiment, the level of thedisplay-attribute setting signal is changed at a specific timing. Forexample, as illustrated in FIG. 13A, in the case where fifteen imagecaptures are performed during a single frame period, the fifteen imagecaptures are divided into three groups, and the level of thedisplay-attribute setting signal is changed for each five imagecaptures. In this embodiment too, the time-series change in the level ofthe common display attribute is a monotonic increase, a monotonicdecrease, or random. In the example illustrated in FIG. 13A, thetime-series change is a monotonic increase.

With such an operation timing, it is possible to determine the trend ofthe whole movement of a subject. In the case of changing the sensitivityfor each image capture in accordance with the level of thedisplay-attribute setting signal, if the image capture is performed manytimes, the sensitivity may no longer have a sufficient difference foreach image capture. Even in such a case, if the level of thedisplay-attribute setting signal is changed in units of groups (e.g.,for each five image captures), as illustrated in FIG. 13B, it ispossible to observe both the detailed movement of the subject in eachgroup and the whole movement in the time base among the groups.

Fourth Embodiment

An imaging device 102 according to a fourth embodiment is different fromthe imaging device 100 according to the first embodiment in that theimaging device 102 includes a display 180. The same parts as those ofthe imaging device 100 will be omitted from the following description,and most of the description refers to different parts.

FIG. 14 is a block diagram schematically illustrating the configurationof the imaging device 102 according to the fourth embodiment.

Examples of the imaging device 102 include a smartphone, a digitalcamera, a video camera, and the like. The imaging device 102 includesthe optical system 110, the image obtaining unit 120, a signalprocessing circuit 160, a system controller 170, and the display 180.The imaging device 102 may further include the external-signal detectingunit 150 described in the second embodiment.

The signal processing circuit 160 is, for example, a DSP. The signalprocessing circuit 160 receives data output from the image obtainingunit 120 and performs a process such as gamma correction, colorinterpolation, space interpolation, or automatic white balance. Thesignal processing circuit 160 has functions corresponding to theabove-described image output unit 130. The signal processing circuit 160outputs multiple-exposure image data, a plurality of separatedindividual image data, image data on which an indicator is superimposed,and the like.

The display 180 is, for example, a liquid crystal display or an organicEL display. The display 180 can display an image on the basis of signalsoutput from the signal processing circuit 160. Examples of the imageinclude the images illustrated in FIGS. 4, 6, 7, 8A, 8B, 9A, 9B, and12B. It is needless to say that the display 180 can display an imageobtained with a typical exposure.

The display 180 may include an input interface such as a touchscreen.Accordingly, by using a stylus and through the input interface, a usercan select the details of a process performed by the signal processingcircuit 160, can control the signal processing circuit 160, and can setconditions for an image capture performed by the image obtaining unit120.

The system controller 170 controls the overall imaging device 102. Thesystem controller 170 is typically a semiconductor integrated circuit,such as a CPU.

According to this embodiment, it is possible to immediately observe acaptured image by the display 180 displaying the captured image and toperform control by using a graphical user interface (GUI) on the display180.

An imaging device according to an embodiment of the present disclosurecan be used for a digital camera, a digital video camera, a cell phonewith a camera, a medical camera such as an electronic endoscope, anonboard camera, a camera for robot, and the like.

What is claimed is:
 1. An imaging device, comprising: a pixel, and acontroller that sets a sensitivity of the pixel according to an externalsignal, wherein the external signal includes sound, vibration orinclination.
 2. The imaging device according to claim 1, furthercomprising: a detection unit that is configured to detect the externalsignal and that is configured to output a signal according to theexternal signal, wherein the controller sets the sensitivity of thepixel according to the signal.
 3. The imaging device according to claim1, wherein the controller sets the sensitivity of the pixel according toa signal output from a detection unit that is configured to detect theexternal signal.
 4. The imaging device according to claim 1, wherein thepixel includes a photoelectric converter including a first electrode, asecond electrode facing the first electrode, and a photoelectricconversion film between the first electrode and the second electrode,and a signal detection circuit electrically connected to the secondelectrode.
 5. The imaging device according to claim 3, wherein apotential difference between the first electrode and the secondelectrode is changed according to the external signal.
 6. The imagingdevice according to claim 5, further comprising a voltage controlcircuit that supplies a voltage to the first electrode, wherein thevoltage control circuit changes the voltage according to the externalsignal.
 7. The imaging device according to claim 1, wherein the externalsignal is emitted from a subject to be captured by the imaging device.8. The imaging device according to claim 1, wherein the sensitivity ofthe pixel is changed in a frame period.
 9. The imaging device accordingto claim 1, wherein the imaging device captures at least two pieces ofimage data with different sensitivities in a frame period.